Cloud and data centers have a greater-than-ever need to satisfy analytics running on big data (e.g., MapReduce, Pregel, message passing interface (MPI), Dryad, etc.). Fast data transmission is needed to deliver good performance, e.g., shuffling accounts for 30% of MapReduce time.
A shared computing cluster imposes further challenges with workloads of mixed priorities and performance goals. For instance, a service job (e.g., an online application) is sensitive to delays, while a batch job (to be executed as a single unit) is sensitive to bandwidth.
Existing approaches include using task schedulers to place tasks close to data (maximizing data locality) to minimize network traffic. These schedulers, however, design a traffic matrix assuming fair sharing of network resources amongst the tasks. Thus there might be policy considerations (such as priorities that can vary over time) which cannot be accounted for with a conventional task scheduler.
Network schedulers schedule flows based on properties of the tasks generating them (such as flow size, deadline, priority, which flows belong to the same task (co-flow), etc.). Essentially, a network scheduler attempts to schedule flows belonging to the same task to make progress together. Therefore, network schedulers can improve the basic transmission control protocol (TCP) by not sharing the network resources fairly.
Schedulers for improving performance of analytics on big data would be desirable.